word2vec Parameter Learning Explained

The word2vec model and application by Mikolov et al. have attracted a great amount of attention in recent two years. The vector representations of words learned by word2vec models have been shown to carry semantic meanings and are useful in various NLP tasks. As an increasing number of researchers would like to experiment with word2vec or similar techniques, I notice that there lacks a material that comprehensively explains the parameter learning process of word embedding models in details, thus preventing researchers that are non-experts in neural networks from understanding the working mechanism of such models. This note provides detailed derivations and explanations of the parameter update equations of the word2vec models, including the original continuous bag-of-word (CBOW) and skip-gram (SG) models, as well as advanced optimization techniques, including hierarchical softmax and negative sampling. Intuitive interpretations of the gradient equations are also provided alongside mathematical derivations. In the appendix, a review on the basics of neuron networks and backpropagation is provided. I also created an interactive demo, wevi, to facilitate the intuitive understanding of the model. 1 Continuous Bag-of-Word Model 1.1 One-word context We start from the simplest version of the continuous bag-of-word model (CBOW) introduced in Mikolov et al. (2013a). We assume that there is only one word considered per context, which means the model will predict one target word given one context word, which is like a bigram model. For readers who are new to neural networks, it is recommended that one go through Appendix A for a quick review of the important concepts and terminologies before proceeding further. Figure 1 shows the network model under the simplified context definition2. In our setting, the vocabulary size is V , and the hidden layer size is N . The units on adjacent An online interactive demo is available at: http://bit.ly/wevi-online. In Figures 1, 2, 3, and the rest of this note, W′ is not the transpose of W, but a different matrix instead. 1 ar X iv :1 41 1. 27 38 v3 [ cs .C L ] 3 0 Ja n 20 16 Input layer Hidden layer Output layer x1 x2 x3 xk